Hybrid Methods for Continuous State Dynamic Programming

We propose a method for solving continuous-state and action stochastic dynamic programs that is a hybrid between the continuous space projection methods introduced by Judd and the discrete space methods introduced by Bellman. Our hybrid approach yields a smooth representation of the value function while preserving the computational simplicity of discrete dynamic programming. The method is especially well suited for implementation in a vector processing environment such as MATLAB or GAUSS, and makes it possible to automate the setup and solution of continuous space dynamic programs in a way that previously seemed elusive. Dynamic programming, while familiar and often the natural way to handle many decision problems, has not seen widespread application. In part this is due to the inherent limitations due to the curse of dimensionality: problems with many state variables become too large to solve. Another reason, however, is surely due to the diiculties one faces in formulating models and solving them. The practice of dynamic programming seems to be the domain of specialists who program each problem anew. This need not be the case. Discrete time dynamic programming problems, for which the state and policy variables can take on a discrete nite set of values, can be completely characterized by a discount factor and two matrices, one describing the current reward function and the other describing the state transition function. Solution methods in this case require maximization over the elements in each column of a matrix and simple matrix arithmetic (addition, multiplication and a linear solve operation). We propose here a similar method for problems with continuous state and action variables that is a hybrid between projection and discretization methods and that retains the best features of both. With this approach one can characterize a DP problem with a discount factor and three matrices, with the addition, for stochastic 1